WORLD INTELLECTUAL PROPERTY ORGANIZATION 
International Bureau 




PCT 

INTERNATIONAL APPLICATION PUBLISHED UNDER THE PATENT COOPERATION TREATY (PCT) 



(51) International Patent Classification 6 : 
G06F 3/02, 3/14 



Al 



(11) International Publication Number: WO 95/32461 

(43) International Publication Date: 30 November 1 995 (30. 1 1 .95) 



(21) International Application Number: PCT/AU95/00300 

(22) International Filing Date: 23 May 1995 (23.05.95) 



(30) Priority Data: 

PM 5806 
PM 7560 



23 May 1994 (23.05.94) AU 
19 August 1994 (19.08.94) AU 



(71) Applicant (for all designated States except US): AUS- 

TRALIAN INSTITUTE OF MARINE SCIENCE [AU/AU]; 
Cape Ferguson, Via Townsville, QLD 4810 (AU). 

(72) Inventor; and 

(75) Inventor/Applicant (for US only): MACDONALD, Bruce, 
William [AU/AU]; 12 Ralston Street, West End, Townsville, 
QLD 4810 (AU). 

(74) Agent: SPRUSON & FERGUSON; G.P.O. Box 3898, Sydney, 
NSW 2001 (AU). 



(81) Designated States: AM, AT, AU, BB, BG, BR, BY, CA, CH, 
CN, CZ, DE, DK, EE, ES, FI, GB, GE, HU, IS, JP, KE, 
KG, KP, KR, KZ, LK, LR, LT, LU, LV, MD, MG, MN, 
MW, MX, NO, NZ, PL, PT, RO, RU, SD, SE, SG, SI, SK, 
TJ, TM, TT, UA, UG, US, UZ, VN, European patent (AT, 
BE, CH, DE, DK, ES, FR, GB, GR, IE, IT, LU, MC, NL, 
PT, SE), OAPI patent (BF, BJ, CF, CG, CI, CM, GA, GN, 
ML, MR, NE, SN, TD, TG), ARIPO patent (KE, MW, SD, 
SZ, UG). 



Published 

With international search report. 



(54) Title: A HUMAN/MACHINE INTERFACE FOR COMPUTING DEVICES 



(57) Abstract 

A computing device having a hu- 
man/machine interface is disclosed. An 
input device (10) has a plurality of keys 
(T, 1, 2, 3, 4) each corresponding to a 
separate digit of the human hand. The 
keys (T, 1,2, 3, 4) are arranged to match 
the relative sequential relation of the dig- 
its. The input device (10) is coupled to a 
processor device (30), in turn coupled to 
a visual display device (50). The proces- 
sor device (30) receives instructions or 
data from the input device (10) and out- 
puts information to be displayed to the 
display device (50). The processor de- 
vice (30) is programmed to cause the vi- 
sual display of indicia (60-70) wherein 
each indicium represents a user initiated 
instruction or data input and is displayed 
in one-to-one correspondence with the se- 
quential relation of one or more of the 
keys (T, 1, 2, 3, 4). 
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A Human/Machine Interface f or Computing Devices 

Field of the Inve ntion 

This invention relates to a human/machine interface for computing devices. 
The interface is particularly, but not exclusively, suited to use with computing devices 
implemented in a compact format, and for computing devices to be used in mobile and 
adverse environments. 

Background of the Invention 

A particular type of human/machine interface is termed a graphical user 
interface, and basically is constituted by software resident in a computing device that 
translates instructions or data to visual information to be displayed on a visual display 
device. 

One simple form of graphical user interface is text-based and utilises cursor 
control to enable navigation about a screen, however the predominant trend in known 
graphical user interfaces is the use of "windowing" systems that operate by 'point and 
click' actions by the user. Examples of windowing software systems are Apple 
QUICKDRAW, Microsoft WINDOWS and XWINDOWS created by MIT. Often 
windowing interfaces also incorporate the earlier form of cursor control, giving the 
user some form of flexibility in operation of the computing device, particularly with 
respect to data input. The icons and windowing structures presented to the user do not, 
however, provide access to the galaxy of commands and operations, so it is always 
necessary for the user to resort to operation of the keys of a conventional qwerty 
keyboard. There thus is the need for the user to learn and remember the qwerty 
keyboard layout. 

Other examples of human/machine interfaces can be obtained from the text 
"Cognitive Aspects of Visual Languages & Visual Interfaces", edited by M.J. Tauber 
et al, published by North-Holland, Amsterdam, 1994. 
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Conventional user input devices such as keyboards, mouses and track balls 
require a benign and stable environment since the user must exhibit fine nervous system 
motor control in the use of the fingers. Mobile environments do not lend themselves to 
such fine movements, and can make operation of computing devices impracticable or 
even impossible. 

The present invention is directed to overcoming or at least ameliorating one or 
more of the disadvantages noted above. Alternatively, or additionally, it can also be 
said to provide a useful alternative to known human/machine interfaces. 

The present invention is directed to a human/machine interface that operates on 
the principle of 'what you see is what you press', and therefore removes the need for 
the user to be familiar with operational commands or the qwerty keyboard layout, 
rather maps input keystrokes directly from the display to the input device. 

The human/machine interface of the present invention is particularly suited to 
use: (a) underwater, where there is a restricted range of human movement; (b) in a 
terrestrial mobile environment, (i.e. particularly 'wearable computers' and in vehicle- 
borne situations) where high levels of motion make it difficult for the user to operate a 
computing device via an input device; and (c) in airborne environments, where there 
are again high levels of motion acting on the user of the computing device. 

It is convenient now to define some of the terminology that is used throughout 
the specification. 

The fingers and thumb of the human hand are referred to as "digits". The term 
"digits" also embraces toes. 

A "key" is to be understood as an input transducer operated by a single digit, 
and can include a pushbutton, switch, force transducer or the like. An "input device" 
typically can have four keys, one for each of the fingers, and one or more "keys" for 
the thumb, given that the thumb has greater dexterity than the fingers. Even so, there 
can be any number of keys in the range 1 to 5. The activation/operation of one or a 
combination of the keys at the same time is referred to as a "chord". A "couplet" is 
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two consecutive chords, or two simultaneous chords performed by the left and right 
hand. 

A "display device" is to be understood a hardware element such as a video 
monitor or liquid crystal display. The display device thus presents the viewer with an 
image that, for convenience, will be termed a "display". 

A display visually presents "indicia". An indicium is presented in a manner 
that represents one or more keys. A number of indicia in close spatial or sequential 
arrangement is termed a "panel". Each panel overlays a number of notional non- 
overlapping sequential regions. In that sense the regions can also be thought of as 
partitioning a portion of the display. An indicium can span over one or more regions in 
the sense that it is oriented substantially orthogonally to the regions. An indicium itself 
can be either contiguous or discontiguous. An indicium also is referred to as a 
"button". 

Summary of the Invention 

Therefore, in one form the invention discloses a computing device having a 
human/machine interface comprising: 

an input device having a plurality of keys each corresponding to a separate 
digit of the human hand, the keys being arranged to match the relative sequential 
relation of the digits; 

a visual display device; and 

a processor device receiving instructions or data from the input device and 
outputting information to the display device, the processor device being programmed to 
cause the visual display of indicia, and wherein each indicium represents a user initiated 
instruction or data input and is displayed in one-to-one correspondence with the 
sequential relation of one or more of the keys. 

Preferably, in use the computing device, a user temporally notionally partitions 
the visual display of the display device into a plurality of non-overlapping regions, each 
region corresponding to one of the digits, and the regions being in the same sequential 
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relation as the digits, and whereby any one or more regions intersected by an indicium 
corresponds to the combination of keys that give effect to the instruction or data input 
represented by the indicium. 

Preferably, the indicia and the regions are arranged orthogonally. 

Preferably, there are five keys, and therefore five notional regions. 

Advantageously, some of the user initiated instructions or data inputs 
correspond to at least two discrete combinations of keys. 

Preferably, the indicia are arranged within a panel, and a number of panels can 
be displayed on the display device. A panel can be coterminous with the corresponding 
regions. 

Preferably, the input device further comprises at least two extra keys, each 
operable by a respective opposite articulation of the thumb with respect to the thumb 
key, each extra key being visually displayed on the display means by a separate 
indicium, such as a thumbline arranged to be either atop a panel or below a panel 
respectively to indicate the direction of articulation of the thumb. 

The invention in another form further discloses a human/machine interface 
program for implementation on a computing device, the computing device including an 
input device having a plurality of keys each corresponding to a separate digit of the 
human hand, the keys being arranged to match the relative sequential relation of the 
digits, a visual display device and a processor device to give effect to the program, the 
program comprising: 

an input element to receive instructions or data from the input device; 

an output element to pass display information to the display device; and 

a processing element, coupled to the input element and the output element, to 
cause the display of indicia, and wherein each indicium represents a user initiated 
instruction or data input and is displayed in one-to-one correspondence with the 
sequential relation of one or more of the keys. 

In a preferred form, some of the user initiated instructions or data inputs 
correspond to at least two discrete combinations of keys. 
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The invention in a yet further form discloses a method for implementing a 
human/machine interface for a computing device, the computing device including an 
input device having a plurality of keys each corresponding to a separate digit of the 
human hand, the keys being arranged to match the relative sequential relation of the 
digits, and a visual display device; the method comprising the steps, implemented on 
processor means, of: 

visually displaying indicia on the display device, each indicium being in one- 
to-one correspondence with the sequential relation of one or more of the keys, thereby 
to allow input of an instruction or data represented by each indicium by the respective 
one or more keys. 

Preferably, in practice of the method a user performs the steps of: 

notional partitioning the visual display device into a plurality of non- 
overlapping regions, each region corresponding to a key and being arranged with the 
same sequential relation as the digits; and 

activating the one or more keys corresponding to the one or more regions 
intersected by an indicium to give effect to the instruction or data input represented by 
the indicium. 

Preferably, there are five keys, and therefore five notional regions. 

Preferably, the method further comprises the steps of displaying two extra 
indicia, each extra indicium corresponding to a shifted state of the user's thumb, thus 
providing for implementation of further instructions or data inputs. 

The invention yet further discloses a method for implementing a 
human/machine interface for a computing device, the computing device including an 
input device having a plurality of keys each corresponding to a separate digit of the 
human hand, the keys being arranged to match the relative sequential relation to the 
digits, and a visual display device; the method comprising the step, implemented on 
processor means, of: 

displaying a regular array of indicia on the display device, each row and 
column of the array respectively being in correspondence, in an orthogonal manner, 
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with the sequential relation of one or more of the keys, and each indicium 
corresponding to a sequence of a row and a column key, thereby to allow input of 
instruction or data represented by each indicium by the respective sequence of keys. 

In a most preferred form the sequence of the keys commences with the thumb. 

Brief Description of the Drawing s 

Embodiments of the invention now will be described with reference to the 
accompanying drawings, in which: 

Figs, la and lb show schematic arrangements of a computing device for 
implementing a man/machine interface; 

Fig. 2 shows a functional block diagram for the computing device of Figs, la 

and lb; 

Fig. 3 shows a display presented to the user in use of the man/machine 
interface; 

Fig. 4 shows another display; 

Fig. 5 shows a pull-out display arrangement; 

Fig. 6 shows a yet further display illustrating shifted states; 

Figs. 7a and 7b show an example of a pop-up display arrangement; 

Figs. 8a, 8b, 8c and 8d show sets of chords for a number of different indicia- 
representations; 

Fig. 9 shows an arrangement for a numerical pad; 

Fig. 10 shows a virtual typewriter; 

Fig. 11 shows a virtual calculator; 

Fig. 12 shows a further display of cross panels; 

Figs. 13, 14, 15a, 15b, 16a and 16b show yet further chord arrangements; 

Figs. 17a-17c show a segmenting cursor arrangement; 

Figs. 18a and 18b show a segmenting pointer arrangement; 

Fig. 19 shows the mechanical arrangement of an input device; 

Figs. 20a and 20b show the mechanical arrangement of a further input device; 
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Fig. 21 shows a conventional qwerty keyboard layout; and 
Figs. 22 to 24 are flow diagrams supporting the methodology of embodiments 
of the invention. 

Description of Preferred Embodiments 

Fig. la shows a schematic arrangement for the human/machine interface 
implemented on a computing device. An input device 10 is coupled with a processing 
device 30, in turn coupled with a display device 50. The input device 10 has five input 
keys which are denoted as 4,3,2,1 and T. As is apparent from the representation of the 
human hand, each of the fingers and the thumb correspond sequentially to the like- 
referenced keys, and any activation of one or a combination of keys produces "a 
chord". Fig. lb shows two thumb keys C,A in addition to the T key that serve a 
special function which presently will be described. 

The processing device 30 can be any off-the-shelf personal computer, although 
for mobile applications should be compact and rugged in nature; for example, in the 
PC/ 104 format. The display device 50 also can be conventional, although again it is 
preferred that it be as compact as possible, and in this respect the PRIVATE EYE (TM) 
device manufactured by Reflection Technology, Inc. is particularly suitable. 

In operation, the processing device 30 causes a display of information on the 
display device 50. In response to that information, the user inputs instructions or data 
by means of keyed chords. Indicia displayed on the display 50 corresponds directly to 
a particular chord, thus giving effect to either the instruction or data input represented 
by the indicium. As noted above, the principle is essentially 'what you see is what you 
press'. 

Fig. 2 shows a functional block diagram of a computing device upon which the 
human/machine interface can be implemented. The input device 10 comprises a multi- 
switch unit 20 constituted by the keys 4-1, T shown in Fig. la. The switch unit 20 is 
connected to a debounce and latch circuit 22 that is configured to output signals only on 
release of the keys, as opposed to on initial depression of the keys. In this way, 
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individual keys in a combination of keys can be depressed separately in time, however 
only the total combination is signalled when all those keys are coincidentally released. 
The output from the debounce and latch circuit 22 passes to a bi-directional parallel 
port 24 of a conventional type, with the output thereof constituting a bit pattern that can 
be decoded by the software resident in the processing device 30. In one preferred form 
the input bit pattern is binary coded leading to a raw decimal value for any chord as a 
combination of key presses. 

In the same way as a conventional keyboard inputs to a personal computer, the 
input bit pattern functions as an interrupt to the operating system. The interrupt 
function is represented by the interrupt handler 32. The interrupts then pass via an 
input parser 34 to a command manager 36. The command manager 36 interacts with 
the host operating system 38, a feedback manager 40 and associated audio handler 42 
and applications software 44. The command manager 36 also co-operates with the 
visual handler 46 which in turn drives the display device 50. 

The audio handler serves a particular function in providing an auditory 
feedback mechanism to the user on activation of any key. A pentatonic scaling is 
preferred, which is the division of one octave into five discrete frequencies. The 
division is not uniform, rather based on a temporal consideration of what sounds 
pleasant to the ear. A chording of more than one key results in the 
reproduction/feedback of the respective notes in combination. Pentatonic scales per se 
are well known. 

The software component of the interface is essentially transparent to the user. 
The user is concerned only with display presented on the display unit 50 as it relates to 
the sequential relation of the keys equating or mapping onto the digits of the hand. 

Fig. 3 shows a display presented on the display device 50. The display, in this 
instance is notionally (temporally) broken to five non-overlapping regions extending as 
bands across the display; the panel constituted by the displayed indicia/buttons 60-70 is 
equivalent in dimension to the whole display. The regions are shown indicated with the 
nomenclature corresponding to the keys/digits of Fig. la and lb. The processing 
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device is not concerned with the notional partitioning, rather simply retains a page table 
of currently-displayed indicia indexed by raw chord value and compares the raw chord 
value input by the user in a look-up routine to give effect to the chorded instruction of 
data input. 

The 'arrow' indicia 60,64 represent commands that can step through operating 
system levels, whilst the 'QUIT' indicium is a command used to terminate a session. 
The other three indicia 66-70 represent commands to implement either a word 
processing package, a "two-up" gaming application or another non-specific applications 
program. 

Each of these regions maps directly onto (equates to) the relative sequential 
arrangement of the keys on the input device 10. That is, there is one-to-one 
correspondence of the sequential relation of one or more of the keys with the sequential 
relation of each indicium. The term sequential has been used, but the one-to-one 
correspondence in most cases can also be thought of as the spatial relation of the keys 
or the order of the keys, where that spatial relation or ordering is itself in one-to-one 
correspondence with the digits. The top-most region corresponds with the position of 
the little finger key 4, and so on to the bottom and the thumb key T. To give effect to 
any one of the indicia, the user selects the combination of keys (i.e. the chord) 
corresponding to the respective indicia. This is assessed by observing which regions 
are intersected by (overlapped, crossed, covered, extended-over, or coincident with) an 
indicium. 

For example, in order to action the 'QUIT 1 command, the user activates 
(chords) the keys "3,2,1" in combination. To give effect to the WP application, the 
user chords key combination "1 & B". TWO UP' is effected by the key combination 
M 2 & 1", and 'APP'N 3' is accessed by the key combination "4 & 3". The user simply 
identifies which regions are intersected by whichever indicium, then chords what is 
observed. 

In this example, the one panel (i.e. the set of indicia), and hence the five 
regions, extends over the whole area of the display. It is equally the case that a panel 
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and the associated regions may not extend over the whole area of the display, so long as 
the relative sequential relation of regions be maintained in one-to-one correspondence 
with the keys. It is equally applicable to arrange the regions to extend vertically of the 
display. In that case, the indicia would be horizontally extending. In essence, indicia 
and regions are orthogonal. It is further possible for the relative position of the thumb 
to be inverted such that it maps to the top of the screen, and therefore the little finger 
maps to the bottom of the screen. In view of these possibilities it is convenient to refer 
to a "thumb origin" to give a sense to the order of the keys to the indicia. In all cases 
the relative sequential relation between the digits/keys and the regions stays the same 
regardless of the specific orientation of the input device 10 or the display device 50. 
As has been previously noted, there may be less than five keys provided. The 
convention adopted is that the sequence always commences with the thumb, hence an 
input device with only three keys corresponds to the thumb and index and middle 
fingers. 

Fig. 4 shows a subsequent display presented when the 'TWO UP' application 
program of Fig. 3 is selected. The display contains a panel 72 containing a number of 
indicia. Each indicium represents a command for use in playing of the TWO UP' coin 
toss game. The panel 72 is notionally divided into five regions, represented in the 
nomenclature previously indicated. Each of the indicia/buttons in the panel 72 is 
activated by a unique chord. The $10 button is operated by the "T/B" single key 
cjiord, the $5 button by the "1" single key chord, the 'CANCEL BET' button by the 
"T/B & 1 " key chord, and the coin toss button by the chord formed by all the keys (ie. 
"T/B & 1 & 2 & 3 & 4"). 

The display also includes a 'hanging-down' panel 74 and a 'hanging-up' panel 
76, each notionally partitioned into five regions that are arranged orthogonally to the 
regions of the other panel 72. These panels relate to specific functions associated with 
the extra thumb keys A & C, shown in Fig. lb that correspond to the articulated thumb 
positions. The extra thumb keys A,C give effect to specific 'shift' functions. The 
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shifted states increase the instruction/data input set from a maximum 31 (2-1 5 ) 
combinations available with only five keys (T and 1-4). 

The C key conveniently has been prescribed a pre-shift function, in that upon 
activation a shifted state is entered whereupon the subsequent chord takes on a different 

5 instruction or data entry value than it would had the C key not been depressed. 

Considering the example of Fig. 4, upon depression of the key C access is obtained to 
the hanging-down panel 74, and the five indicia contained therein. In this case, the five 
indicia each correspond with one region alone. It is, of course, possible to have any of 
the indicium extending over more than one region. 

10 Another type of shift function is termed 'shift immediate' . This is effected by 

the A key. This situation is somewhat akin to the shift key to be found on a 
conventional keyboard, in that the key A forms an alternative to the key B when used 
alone, and also can be used in combination with the other keys 1-4. The hanging-up 
panel 76 is accessed by use of the A key in combination with keys 1-4. 

15 Fig. 5 shows a section of the display of Fig. 4, particularly indicating the 

hanging-down pre-shift panel 74. Once the pre-shift key C is effected, the subsequent 
chord will give effect to one of the indicia. In the present example, the subsequent 
combination is the sequence of "3" key alone (being the EDIT function as highlighted). 
This results in a further panel 78 being 'pulled-out', and any subsequent chord gives 

20 accesses each one of the five indicia indicated in the new panel 78. Again, it is merely 
for the sake of convenience that the indicia corresponds one-to-one with the five 
regions. 

Fig. 6 shows further arrangements for the representation of shifted states. 
Three 'floating' panels 80,82,84 are shown within the display 50. Each of these panels 
25 contains a number of indicia (not labelled), with each panel being notionally divided 
into five regions, as indicated, that sequentially map onto the keys. The indicia of the 
panel 80 are accessed without the use of either of the shift keys A,C. The panel 82 
includes a 'thumbline' 88 (or 'baseline'), represented by the thickened line to the 
bottom of the panel. This thumbline 88 provides a visual indication that a shifted state 
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is required in order to access the indicia within the panel 82. The thumbline 88 being 
at the bottom of the panel indicates the need to depress the A key. In other words, a 
'shift immediate' function will be effected as previously described. For the shift 
immediate function, there will only be four notional regions within the panel, as 
indicated. This is for reason of the thumb being used to activate the A key to enter the 
shift immediate state, thus leaving only the remaining four digits to be available for 
use. There is a certain logic to the thumbline arrangement, in that the thumb must be 
articulated downwardly away from the usual B key to activate the A key, and its 
downward sense is consistent with the thumbline 88 being to the bottom of the panel 
82. The remaining floating panel 84 has a thumbline 90 located to the top. This 
indicates the need to articulate the thumb upwardly to operate the C key for a pre-shift 
state. In the pre-shift state there are five notional regions, as noted. 

The display also includes a further panel 86 as an inverted L-shape in the top 
left hand corner of the display. The 'hanging-down' convention does not apply to this 
panel 86, rather the thumbline principle is in effect. The panel 86 is shown as a single 
panel, however contains two discrete sets of notional regions, hence more properly is 
two panels. The indicia appearing down the left hand side of the display are tied to the 
thumbline 92 appearing at the bottom edge thereof, and is notionally broken into the 
notional regions as indicated. Since the thumbline 92 is to the bottom of this part of the 
panel, it represents use of the A key, in which case there are only four regions as 
previously described. The indicia extending across the top edge of the display are tied 
to the other thumbline 94. This thumbline is again to the bottom edge of the panel, 
hence again represents use of the A key. 

Fig. 7a shows a panel 96 incorporating five indicia. The notional regions 
corresponding to the keys also shown. If the "1" key is chorded, a further panel 98 
pops-up. The further panel 98 contains five different indicia. As can be noted, the 
regions within the frame are orthogonal to those for the frame corresponding to the 
parent panel 96. A similar pop-up effect would occur for the other indicia of the first 
panel 96. 
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Figs. 8a and 8b show the complete set (or 'galaxy') of chords for a five-button 
input device. The dashed lines represent the boundary of five notional regions within 
which the indicia are bounded. The indicia are shaded, and for convenience are as a 
type termed "bar chords". Note that a number of the bar chords are not formed by a 
contiguous key press and these special chords are termed "hollow bar chords". Also 
shown beside each indicium is its "raw chord value". By a convention, the thumb 
origin is at the left hand side, and the notional regions are represented in a binary form 
so that any chord also can be represented as a unique decimal number - this is useful in 
an implementation of an embodiment of the invention as will become apparent. The 
binary convention is reversed, increasing from 0-31 from right to left, but equally could 
be increasing 0-31 from left to right. 

Fig. 8c shows five different style of chord representations, for, as it turns out, 
the chords having the decimal values 1-5. The first (left-most) column shows bar 
chords for which the partitioning into five regions has to be visualised without a 
graphical prompt - compare this with the bar chords in the third column. In the second 
column, the boundary of the five regions is represented by the bottom horizontal line, 
and the key(s) to be chorded by the vertical line marks. This representation is termed 
"glyph chords". The fourth column represents the vertical, rather than horizontal 
orientation of bar chords, with the thumb origin occurring at the bottom. The fifth 
column representation of dots and line marks are termed "dot chords". 

Fig. 8d shows the galaxy of glyph chords together with their raw chord value. 

Fig. 9 shows a panel 104 in the form of a numeric keypad. All of the digits 0- 
9 are represented. The thumb origin is in the bottom left hand corner of the " 1 " 
indicium. The panel 104 is broken up into two parts, the first part being the 3 x 3 
array 106 of the numerals 1-9, and the second part being the button 108 for the 0 
numeral. The "0" button is activated by chording "T & 1 & 2". The other numerals 
are accessed by means of chording a "couplet" . That is, in the 3 x 3 array each 
particular numeral button requires a separate sequential combination of two chords for 
it to be accessed. For example, the "1" button is accessed by the couplet "T" & M T" , 



WO 95/32461 



PCT/AU95/00300 



-14- 

whilst the "6" button is accessed by the couplet "1" & "3". Couplet chording is 
somewhat equivalent to the pop-up menus previously discussed, in as much as the 
popped-up buttons are always displayed rather than being hidden. 

Fig. 10 shows a form of virtual typewriter, basically comprising two panels 
110,112 side-by-side. The partition between the panels is represented by the vertical- 
line. The panel 110 comprises a 5 x 5 array of 25 buttons representing 25 letters of the 
roman alphabet. Activation of the respective letter buttons is by couplet chording. The 
thumb origin for the panel 110 is at the bottom left hand corner of the "e" button. 

By way of example, the letter "r" button is activated by chording "2" & "2", 
and the "j" button is activated by chording "3" & "4". The "Space" button is activated 
by chording "T &1 & 2 & 3 & 4". The spatial arrangement of the letter buttons 
contemplates the relative frequency of occurrence of letters in the English language and 
maps that frequency onto the dexterity of the thumb and fingers. Thus, the vowels 
a,e,i, and o are accessed by combinations of the thumb and first finger that are the most 
dexterous digits. 

The panel 112 compliments the typical word processing functions. The thumb 
origin for the panel 112 is at the bottom left hand corner of the "Period" button. This 
panel also operates on the basis of couplet chording. Thus the "Tab" button is accessed 
by the chord "T & 1 & 2" & "2". The "Shift" button is accessed by the chord "3 & 4" 
& "T". The "Numbers" and "Symbols" buttons are in the nature of pop-out menus 
giving access to, respectively, a numerical panel such as the panel 104 shown in Fig. 9 
and other couplet panel arrays in respect of symbol sets. 

It will be noted that each of the couplet chords contained within both of the 
panels 110,112 is unique from any other. The right-hand panel could be chorded by 
the user's other hand if a second input device were provided. The arrangement shown 
in Fig. 10 is well suited for word processing functions, and can enable touch typing 
much in the manner of a conventional qwerty keyboard with a minimum of tuition. 

Fig. 11 shows a form of a virtual calculator 114. The calculator comprises a 
3x3 couplet pad array 106 (as previously shown in Fig. 9), for which the thumb origin 
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is at the lower left hand corner of the "1" button. The calculator 114 also comprises a 
number of single cord buttons delineated from the couplet pad array 106 as shown by 
the dashed lines. The thumb origin for the chord buttons is the left hand side of the 
panel 116. The top window 118 represents an inner display to indicate the function of 
the calculator 114 much in the nature of a conventional hand-held calculator. The 
single chord required to activate any of the function buttons or "0" button are 
independent of the first chord of any couplet pertaining to the buttons within the pad 
array 106. By way of example, the number 643 is chorded by the sequence "1" & "2", 
"1" & "T" and "T" & "2". This number can be operated on, say to be multiplied by 2 
by the successing chords: "4" then "T" & "1" then "T & 1 & 2 & 3 & 4". 

Fig. 12 shows a display arrangement termed cross-chords. The thumb origin 
for the shaded buttons extending in the upper half of the display along the left hand side 
and horizontally across the mid-point of the display is illustrated as T^ The thumb 
origin for the buttons arranged on the right hand side of the lower half of the display 50 
is shown as T 2 . Fig. 12 also shows the chord required to activate each button, from 
which it can be seen that each chord is mutually exclusive from any other. 

Fig. 13 shows an alternative representation of an indicium 130. The 
convention in this case is that because only the four vertices of the rectangular box are 
displayed, then the thumb always forms a part of the chord. Accepting the further 
convention of the bottom left-hand vertice being the origin, then the chord would be 
formed by the combination "T & 3 & 4". The dot chord representation also is shown. 

Fig. 14 shows an example of the combination of two indicia 132,134 forming 
bar chords and couplets. The two possible bar chords have their dot chord value 
superimposed on them, and respectively are formed by "T" and "1 & 2 & 3 & 4". The 
second indicium 134 also can be utilised in the form of a couplet in conjunction with 
the indicia 136-144 for the five bar chords superimposed on it. Note too that the dot 
chord representation corresponding to each chord has been shown. To activate the 
middle indicium 140 on the background indicium 134, the couplet chording sequence is 
"1 & 2&3&4 1 ' and "3". 
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Fig. 15a and 15b show an alternative representation for couplet chording for a 
number of indicia 150-156. Also shown is the corresponding dot chord representation. 
The convention adopted is that horizontal takes precedence over vertical, thus in the 
case of Fig. 15a leading to the couplet "4" and "4", and for Fig. 15b M 3 & 4" and "T & 
4 M . 

Fig. 16a shows an abbreviated form of an indicium 160 representing three bar 
chords. The two tabs 162,164 located along the bottom edge of the indicium 160 
indicate that "T & 1" are required to access any one of the three superimposed sub- 
indicia 166-170. The dot chord representation for those three sub-indicia 166-170 also 
are shown. In Fig. 16b, the tabs 182, 184 are located to the right hand side of the 
indicium 180, indicating the keys M 3 & 4" are required as well as the corresponding key 
of the superimposed sub-indicia 186-190 to effect a chord. 

A library of conventions can be chosen, and as follows from the example 
presented herein, it is apparent that the left hand side and/or bottom of an indicium 
represents the thumb origin. The thumb is always the starting point - this is 
particularly important if less than five keys are provided. Horizontal takes precedence 
over vertical as does clockwise over anticlockwise. All indicium must satisfy the rule 
that they are resolvable into a 'press path*. 

Embodiments of the present invention have immediate application in relation to 
conventionally-understood computing devices, but equally could be incorporated into 
viewfinder control systems (e.g. camcorders), autotellers, fax and photocopy machines, 
embedded interfaces (e.g. vending machine maintenance interfaces), mobile telephones 
and pagers, portable machinery (e.g. service equipment), video and television remote 
controllers, interactive television controllers, personal stereos, household appliances 
(e.g. clock radios), vehicleonics (e.g. air conditioning, cruise control, stereo, cabin 
controls, etc.), powertool interfaces (e.g. pistol grip-style drills) control mounted (e.g. 
steering wheel or joystick), game and toy interfaces, electronic slate-style magazines, 
pocket organisers, hand-held computing devices, notebook and pen computers. 
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A particular advantage accrues by virtue of the present invention in that the 
indicia can be easily scaled and yet retain the information content thereof. This is in 
contrast conventional windowing-type systems, where icons, if they were able to be 
shrunk, would lose their information content, and also would become difficult to access 
by means of a pointing device. Indeed, prior art windowing systems do not have 
provision for the scaling of a "window" or icons within a window, rather simply 
collapse the window itself. By the use of indicia in accordance with the present 
invention, no such pointing is required, rather the information content will be retained 
so long as the eye can resolve an indicium, of course depending also upon the pixel 
resolution of the display device. This means that a greater number of indicia can be 
included on a display than for the conventional windowing systems. 

Figs. 17a-17c show a display on which a pictorial representation, in this case 
being of a woman, is shown. The display equally could be of a map or an integrated 
circuit layout, for example. The display 50 shows the 5 x 5 array of regions occupying 
the whole of the display. Utilising the couplet chording principle discussed earlier, a 
segmenting cursor or "zoom" function can be achieved. 

The thumb origin is located at the bottom left hand corner of the display 50, 
and in the examples shown, the first chord of the couplet is "2" that results in the 
highlighted representation shown in Fig. 17b. On activation of the other half of the 
couplet chord "3", that particular chorded button is exploded and occupies the whole of 
the display 50 as shown in Fig. 17c. By further couplet chording, yet greater resolved 
detail can be obtained. 

Figs. 18a and 18b show a variation of a segmented display, being a pointing 
function. By keying the couplet chord "3" & "2", that particular button has a further 
segmented array superimposed on it. A further chorded couplet can yet further divide 
one of the buttons within the highlighted button to point more specifically resolved on 
an object displayed. 

Fig. 19 shows a mechanical arrangement for the input device 10. The body 11 
is sized to fit into the palm of the hand of a user, and in the configuration shown is 
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suited to use by the left hand. The thumb therefore wraps around the side of the body 
1 1 whilst the four fingers wrap across the top. The digits therefore can activate the 
keys in a manner previously described. Such an arrangement provides the advantage of 
being useable in a mobile configuration. The body 11 is securely grasped in the palm 
of the hand, and the digits do not have to move other than in a gross closing motion to 
activate the keys. The configuration shown is easily adapted for use by the right hand 
simply by flipping it over, therefore is completely ambidextrous. 

Figs. 20a and 20b show an alternative arrangement for an input device 10'. 
The hand-held device 10' comprises two body portions 12,13 hingedly connected 
together. The arrangement of the keys with respect to each of the body parts 12,13 is 
shown. 

The particular advantage of this configuration is that it is ambidextrous, and 
also can be used either in the manner shown for the controller 10 of Fig. 19 when 
grasped in the palm of the hand, in which case the respective body parts 12,13 are 
arranged at an acute angle to each other, or with the body parts in a common plane for 
use as an on-bench keyboard. 

Fig. 21 shows a conventional qwerty keyboard layout. A further embodiment 
of the invention provides for combined chording/qwerty operation by means of such a 
conventional qwerty keyboard. Thus the keyboard shown in Fig. 21 can be operated in 
the usual manner, however the software resident on the associated processing device 
can recognise when chords are keyed in distinction to single key presses for 
conventional inputs. In the embodiment, the "home" keys and the space bar are used as 
the keys for chording operation. If the right hand is being used for chording, then the 
sequence of qwerty keyboard keys corresponding to the sequence of the digits 
commencing with the thumb is: "space bar, J,K,L,;". If the left hand is being used for 
chording, then the chording keys are: "space bar, F,D,S,A". In operation of software 
embodying this concept, the display presented to the user can be, say, a conventional 
word processing package, with superimposed chording indicia. The only restriction is 
that the so-called "singleton" chords (i.e. the single digit presses) that are excluded. 
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It would be readily apparent to one skilled in programming arts how to write 
code to implement the embodiments of the invention hereinbefore described without the 
exercise of any inventive faculty. In this connection Figs. 22 to 24 are included that 
respectively are flow diagrams of the chording methodology, the couplet methodology 
and the chord/qwerty methodology. In Fig. 22 step 200 detects a key press. If, in step 
202, a chord is to be formed on release, then step 204 checks to determine whether all 
the keys are depressed, and if so step 206 sends the value of the chord keyed. If the 
chord is not to be formed on release, step 208 delays sending of the chord keyed. 

Fig. 23 concerns the extension of Fig. 22 to the forming of couplets. Step 210 
represents waiting for a chord press, and once such is received, step 212 determines 
whether the chord pressed is part of the present couplet. If not, there is a return to step 
210. If the result of step 212 is yes, then step 214 tests whether for the present couplet 
the second part is being awaited. If no, step 216 determines whether it is the first part, 
and if no again, step 218 ignores the chord and returns to step 210 to await a further 
chord, If, however, it is the first part of the chord, then step 220 sets the "waiting" 
flag and returns to step 210. In step 214, if yes, step 220 determines whether the chord 
is the second part, and if not loops to step 210, but if so, step 222 clears the "waiting" 
flag and forwards the couplet press. 

Fig. 24 represents the combined chording/qwerty operation. Step 230 awaits 
the next press, and on receipt of such tests whether chording is allowed in step 232. If 
no, the key press is forwarded for its qwerty value by step 234. If chording is allowed, 
then step 236 determines whether the value is for a chord key (i.e. the home keys as 
described). If not, step 238 determines whether operation is for chording only, and if 
so, step 240 therefore ignores the key press, and otherwise the key pressed value is 
forwarded in step 242 for its usual qwerty value. If, in step 236, the key press value is 
for a chord, step 242 forms the chord. There then follows a determination in step 242 
of whether the chord is a singleton chord, and if not, step 244 sends the chord pressed 
value, but if so, step 246 determines whether singleton chords are allowable, and if not, 
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step 248 forwards the nominal qwerty key value. If singletons are allowed, then step 
250 sends the chord pressed value. 

The following coding is included for illustrative purposes, and is consistent 
with the flow diagrams of Figs. 22 to 24. The coding is written to run on the program 
5 ToolBook (TM) version 3.0, which is a Windows (TM)-based multi-media authoring 
tool published by Asymetrix Corporation of the United States. The code is commented, 
as indicated by the prefix 



to set KSQwertyOn to offOn 

1 o system QwertyOn 

set QwertyOn to offOn 
end 

to set KSremainderOn to offOn 

1 5 system remainderOn -allows non-chordal KBD presses 

set remainderOn to offOn 
end 

to set KSChordingOn to offOn 

20 system ChordingOn -Chording master on/off switch 

set ChordingOn to offOn 
end 

to set KSQwertyOnly to offOn 
25 system QwertyOnly —locks out chords 

set QwertyOnly to offOn 
end 



-allow external objects to change KordOS 
—internal settings 
-this setting allows normal KBD 



to set KSPressReleaseType to Pval 
30 system pressReleaseType -chord formed instantly or on release 

set PressReleaseType to Pval 
end 



to handle Keydown vkey -handle the ToolBook (TBK) message 

35 —message Keydown ie. a KBD key 

system remainderOn -declare system vars for this handler 

system QwertyOn 
system QwertyOnly 
40 system ChordingOn 

system pressRelease Type 
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get ChordingOn 



—check if on, skip if not 



if it = "false" 

forward 

break 

end 



then 



system chordList[31] 



—initialise left & right hands 



10 



15 



20 



25 



30 



35 



40 



RT 




keyspace 


Rl 




keyj 


R2 




keyk 


R3 




keyl 


R4 




keysemicolon 


LT 




keyspace 


LI 




keyf 


L2 




keyd 


L3 




keys 


L4 




keya 



vvkey = lowercase(vkey) -filter upper case of message parameter 

if vvkey = RT or VVkey = Rl or VVkey = R2 or vvkey = R3 or vvkey = R4\ 
or vvkey = LT or vvkey = LI or vvkey = L2 or vvkey = L3 or vvkey = L4 then 

-check if its a Kording key on KBD 

set chord to 0 —reset local variables 

set thumbPressed to false 

set firstPressed to false 

set secondPressed to false 

set thirdPressed to false 

set fourthPressed to false 

if pressReleaseType = "WaitForRelease" then -check how the chord is made 
do 

if keyState(RT) is "down" or keystate(LT) is "down" -keyStateQ is aTBK 
func'n 

set thumbPressed to true 
end 

if keyState(Rl) is "down" or keystate(Ll) is "down" 

set firstPressed to true 

end 

if keyState(R2) is "down" or keystate(L2) is "down" 

set secondPressed to true 

end 

if keyState(R3) is "down" or keystate(L3) is "down" 

set thirdPressed to true 

end 

if keyState(R4) is "down" or keystate(L4) is "down" 
set fourthPressed to true 
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end 

until keyState(RT) is "up" and keyState(Rl) is "up"\ 
and keyState(R2) is "up" and keyState(R3) is "up"\ 
and keyState(R4) is "up" 

5 

if qwertyon = 'false' 

get flushmessagequeueO -clear the KBD buffer 

end 

end 

10 

if pressReleaseType = "instant" then -the other way of chording 
pause 15 — debounce other keys 

if keyState(RT) is "down" or keystate(LT) is "down" 
set thumbPressed to true 
15 end 

if keyState(Rl) is "down" or keystate(Ll) is "down" 

set firstPressed to true 

end 

if keyState(R2) is "down" or keystate(L2) is "down" 
20 set secondPressed to true 

end 

if keyState(R3) is "down" or keystate(L3) is "down" 

set thirdPressed to true 

end 

25 if keyState(R4) is "down" or keystate(L4) is "down" 

set fourthPressed to true 
end 

if qwertyon = "false" 
get flushmessagequeueO 
30 end 
end 

if fourthPressed is true —build raw chord value 

set chord to 1 
35 end 

if thirdPressed is true 
chord = chord + 2 
end 

if secondPressed is true 
40 chord = chord + 4 
end 

if firstPressed is true 
chord = chord + 8 
end 

45 if thumbPressed is true 
chord = chord + 16 
end 
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get flushmessagequeueO 

break 

end 
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— internal error trap 



10 



if qwertyon = 'true' then —forward singleton chords 

if chord = 1 or chord = 2 or chord = 4 or chord = 8 or chord = 16 then 

forward 

break 

end 

end 



get flushMessagequeueO 
chordval = chordlist[chord] 
1 5 send kordPressed chordval to this Page 



20 



else 



if remainderOn = "true" then 

forward 

else 

get flushMessagequeueO 
25 end 
end 
end 

to handle enterbook 

30 send resetKords 
end 



-clear KBD buffer 
—look up the content of raw chord 
—send message to graphic Page 
-a TBK structure 
—page forwards message to all 
—objects on it 
-end of test for non-finger chars 
-below we handle 
-they want all non-chord values 



-another obsessive clear 



-initialise this system book on startup 
-sends message resetKords 



to handle resetKords —set the raw chord contents 

—put other Chords = " "here 
35 Handchords — 

"1,2,3 ,4,5,6,7,8,9,10,11,12,13,14,15, 16,17, 18,19,20,21,22,23,24,25 ,25,27,28,29,30, 
31" 



40 



45 



dotChords = 

"...J....I H...I U,.JL...IH,.I LJ,.LL,.LH.-1L...IIJ.JIL,JIIIJ I...I,U.,UI,LL, 1.1.1,1. 

IJIIJL..JLJJLL,IIJIJIL.JILI,!iil.,lllll" 

system chordList[31] 

fill chordlist with handChords in [item] order 



system remainderOn 
system QwertyOn 
system QwertyOnly 



-declare system vars for this handler 
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system ChordingOn 
system PressReleaseType 

set qwertyOn to "false" --reset to known state 

5 set remainderOn to "true" 
set QwertyOnly to "false" 
set ChordingOn to "true" 
set PressReleaseType to "WaitForRelease" 
end 

10 

—this is a typical button handler, it receives the message from the page 

—this script is contained 

to handle Kordpressed kordval 

if kordval = myKordVal of self -check the parameter of the message 
15 -my predefined attribute, ie. is it me? 

--**** perform some action 

end 
20 end 

to handle buttonClick —allows mouse clicks 

send Kordpressed myKordVal to self -trick me that a key is pressed 
end 

25 

Emboiments of the present invention have industrial applicability at least in 
terms of rendering useful computing devices adapted for portability or use in adverse 
high motion environments. 
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CLAIMS: 

1. A computing device having a human/machine interface comprising: 
an input device having a plurality of keys each corresponding to a separate 

digit of the human hand, the keys being arranged to match the relative sequential 
relation of the digits; 

a visual display device; and 

a processor device receiving instructions or data from the input device and 
outputting information to the display device, the processor device being programmed to 
cause the visual display of indicia, and wherein each indicium represents a user initiated 
instruction or data input and is displayed in one-to-one correspondence with the 
sequential relation of one or more of the keys. 

2. A computing device as claimed in claim 1, wherein, in use the 
computing device, a user temporally notionally partitions the visual display of the 
display device into a plurality of non-overlapping regions, each region corresponding to 
one of the digits, and the regions being in the same sequential relation as the digits, and 
whereby any one or more regions intersected by an indicium corresponds to the 
combination of keys that give effect to the instruction or data input represented by the 
indicium. 

3. A computing device as claimed in claim 2, wherein the indicia and the 
regions are arranged orthogonally. 

4. A computing device as claimed in any one of the preceding claims, 
wherein there are five keys. 

5. A computing device as claimed in any one of the preceding claims, 
wherein some of the user initiated instructions or data inputs correspond to at least two 
discrete combinations of keys. 
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6. A human/machine interface program for implementation on a 
computing device, the computing device including an input device having a plurality of 
keys each corresponding to a separate digit of the human hand, the keys being arranged 
to match the relative sequential relation of the digits, a visual display device and a 
processor device to give effect to the program, the program comprising: 

an input element to receive instructions or data from the input device; 
an output element to pass display information to the display device; and 
a processing element, coupled to the input element and the output element, to 
cause the display of indicia, and wherein each indicium represents user initiated 
instructions or data and is displayed in one-to-one correspondence with the sequential 
relation of one or more of the keys. 

7. An interface programme as claimed in claim 6, wherein some of the 
user initiated instructions or data inputs correspond to at least two discrete 
combinations of keys. 

8. A method for implementing a human/machine interface for a 
computing device, the computing device including an input device having a plurality of 
keys each corresponding to a separate digit of the human hand, the keys being arranged 
to match the relative sequential relation of the digits, and a visual display device; the 
method comprising the step, implemented on processor means, of: 

visually displaying indicia on the display device, each indicium being in one r 
to-one correspondence with the sequential relation of one or more of the keys, thereby 
to allow input of an instruction or data represented by each indicium by the respective 
one or more keys. 

9. A method as claimed in claim 8, wherein, in practice of the method, a 
user performs the steps of: 
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notional partitioning the visual display device into a plurality of non- 
overlapping regions, each region corresponding to a key and being arranged with the 
same sequential relation as the digits; and 

activating the one or more keys corresponding to the one or more regions 
intersected by an indicium to give effect to the instruction or data input represented by 
the indicium. 

10. A method as claimed in claim 8 or claim 9, wherein two extra indicia 
are displayed, each extra indicium corresponding to a shifted state of the user's thumb 
to provide for implementation or further instructions or data inputs. 

11. A method for implementing a human/machine interface for a 
computing device, the computing device including an input device having a plurality of 
keys each corresponding to a separate digit of the human hand, the keys being arranged 
to match the relative sequential relation to the digits, and a visual display device; the 
method comprising the step, implemented on processor means, of: 

displaying a regular array of indicia on the display device, each row and 
column of the array respectively being in correspondence, in an orthogonal manner, 
with the sequential relation of one or more of the keys, and each indicium 
corresponding to a sequence of a row and a column key, thereby to allow input of an 
instruction or data represented by each indicium by the respective sequence of keys. 
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